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Rating*  have  wide  use  In  the  assessment  of  Job  performance  in  a 
variety  of” military  situations.  Tbqr  provide  a  means  of  obtaining  per¬ 
formance  Information  on  large  numbers  of  men  ever  a  broad  range  ef 
activities  In  a  relatively  economical  meaner.  (In  order  to  neke  maximal 
use  of  current  rating  Information  and  to  l^rove  the  quality  of  future 
ratings,  more  Information  about  factors  in  Job  performance  ratlnga  la 
needed.  ^Tfce  purpose  of  the  present  study  was  to  utilise  existing  ratings 
te  obtain  partial  information  about  acme  of  these  factors.  The  study 
was  pointed  tewerd  four  specific  objectives: j 

*  to  determine  the  extent  of  differences  In  ratings  by  super-risers 
and  peers . 

2.  to  estimate  the  structure  of  ratings  sn  several  aspects  of 
performance 

3.  to  estimate  the  relationship  between  ratings  and  (a)  frequency 

of  observation  by  the  rater  and  (b)  length  of  time  rater  bad  worked 
with  or  supervised  ratee  ^ 

4.  to  estimate  the  relatlenshlp  between  ratings  and  background  and 
attltudlnal  variables  , 


The  ratee  group  consisted  of  ^55  BA  personnel  from  samples  obtained 
for  the  follow-up  study  of  reenlistment  intention  (project  NCT  b-lb). 

All  cases  had  been  In  the  Any  approximately  one  year  at  the  time  of  the 
ratings  and  were  assigned  to  a  wide  range  of  combat  and  technical  NOS. 
All  ratees  were  single  Caucasian  males  In  their  late  teens  or  early 
twenties.  They  had  a  mean  ACB  verbal  standard  score  of  111.13  and  6l£ 
had  completed  high  school. 


ratdws 

Each  ratee  was  rated  by  his  Imssdlate  supervisor  and  two  peers  on  a 
17 -Item  scale  (IT  3723)*  The  Items  covered  social,  proficiency,  protoc- 
tivity,  and  military  aspects  of  the  man's  performance,  as  well  as  overall 
evaluations.  For  each  Item,  three  to  six  graded  response  categories 
were  provided.  The  scale  Is  given  In  %psnd1v  A. 

In  general,  raters  Indicated  dally  observation  of  the  ratees* 
performance  (Item  19)  end  had  worked  with  «r  supervised  the  man  for 
appraodaately  six  months  (Item  20).  > 


VARIABLES 


1-19.  Supervisor  responses  to  items  in  FT  3725 

1-17  •  rating  items 

19.  frequency  of  observation 

20.  length  of  time  rater  supervised  ratee 

20-38.  Peer  responses  (Group  P-l)  to  items  in  PT  3725 

1-17*  rating  items 

19.  frequency  of  observation 

20.  length  of  time  rater  worked  with  ratee 

39-57.  Peer  responses  (Group  P-2)  to  items  in  PT  3725 

1-17.  rating  items 

19.  frequency  of  observation 

20.  length  of  time  rater  worked  with  ratee 

58.  Year  of  birth  (Item  %  PT  372*0 

59.  Reenlistment  Intention.  Item  17,  FT  3J2k 

60.  Attitude  toward  Army.  Sum  of  items  4l  through  53,  PT  3724: 
Agree  ■  2,  ?  ■  1,  Disagree  ■  0. 

61.  Education.  A  dichotomy  of  completed  high  school  (code  l) 
versus  nonco^letion  (Code  0). 

62.  ACB  Verbal  Test  score.  Standard  score  from  administration 
of  ACB  at  entry  into  service. 

63.  Rural -Urban  Key.  An  8-ltem  key  from  FT  3426,  administered 
at  entry  into  service. 

64.  Socio-economic  Key.  A  37- item  key  from  PT  3426,  administered 
at  entry  into  service.  Larger  scores  indicate  higher  socio¬ 
economic  level. 


The  Mis  of  ratings  for  each  nan  obtained  fin  the  two  peers  were 
assigned,  one  to  rater  grag  P-1,  and  one  to  rater  group  P-2,  In  a  aon- 
sy  st  ms  tic  nanner.  All  variables  were  lnteroorrelated  (product -Bc—nt- - 
uncorrected  for  course  grouping,  polnt-bi serial,  or  phi  coefficients, 
depending  upon  nuaber  of  scoring  categories)  and  naans  and  standard 
deviations  were  obtained.  A  54- variable  sub-aatrix  of  rating  ltcBS 
(1-17  rated  by  supervisor  and  1-17  by  one  peer  rater  grasp)  was  factored 
by  successive  application  of  a  technique  described  by  Wherry  (1999). 
Rater  group  P-1  was  chosen  as  the  peer  group  to  be  used  la  the  factor 
analysis  on  the  basis  of  the  flip  of  a  coin.  All  supervisor  and  all 
peer  (P-1)  ratings  ware  used  as  clusters  and  ths  factorisation  carried 
out.  A  residual  aatrix  was  obtained  and  factored  by  the  ease  asthod 
(different  Initial  dusters).  The  aethod  yielded  a  hlererehlal  solution 
and  no  rotation  was  done.  The  lntercorrelatlon  aatrix  is  given  in 
Table  1,  and  the  factor  aatrix  In  Table  2,  Appendix  B. 


RESULTS  AID  DdSCOSSXOI 


The  aean  and  standard  deviation  of  each  rating  ltea  for  each  rater 
group  are  given  in  Table  1.  In  general,  the  differences  between  peer 
and  supervisor  groups  appeared  no  greater  than  differences  between  the 
two  peer  groups.  In  terns  of  level  and  variation  of  ratings,  there  was 
no  real  evidence  on  which  to  posit  supervisor-peer  differences.  Differ¬ 
ences  In  the  pattern  of  coefficients  representing  aagnitudes  of  relation- 
ahlp  aaoug  rating  Itcas  for  the  three  rater  grape  were  not  subjected 
to  statistical  analysis.  Vfithln-grap  Intercorrelations  were  unlfosnly 
larger  than  corresponding  betwsen-group  coefficients,  but  this  was  to 
be  expected  since  within  a  given  groip  for  a  given  rates  all  ratings 
ltsas  were  responded  to  by  one  rater.  It  would  be  unlikely  that  inter- 
rater  sgrsMsnt  would  be  greater  than  intra-rater  agreeasnt.  Proa  the 
factor  analysis,  ths  seas  dlasnslons  appeared  to  characterise  both 
supervisor  and  peer  ratios. 


The  lateroorrelatlon  and  factor  astriees  are  given  in  Appendix  B. 
Haven  orthogonal  factors  were  Obtained  froa  the  ^-variable  aatrix  of 
tad  peer  ratings.  Za  general,  the  rating  ltea  laterrelatlon- 
to  be  a  function  of  an  overall  global  evaluation  (Factors  I, 
ZI,  and  IH),  plus  evaluations  on  dlasnslons  of  Job  knowledge  (factors  IV, 
V,  and  VI),  allitaxy  behavior  (fhctors  VII,  vm,  aad  IX),  and  inter¬ 
personal  relations  (factor  X).  factor  XI  picks  «p  loading  fron  both 
peer  aad  supervisor  ratings  on  how  isportaat  the  aan  is  to  his  unit. 

Six  of  ths  eleven  factors  reflect  the  greater  consistency  within  rater 
type  than  between  rater  type. 
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Table  3 


RATINGS  VS  OTHER  VARIABLES 


Table  2.  Coefficient*  were  uniformly  low.  Relationship*  between 
frequency  of  observation  and  ratings  tended  to  be  of  somewhat  greater 
magnitude  for  the  peer  rater  group*.  Since  the  coefficients  were  almost 
all  positive  across  rating  items,  they  may  reflect  some  tendency  on  the 
part  of  peer  raters  to  up-grade  ratees  with  wham  they  work  more  closely. 

Background  and  attitudinal  variable*.  Coefficients  representing 
relationship*  among  background  and  attitudinal  variables  and  ratings  are 
given  in  Table  3«  Age  and  education  yielded  consistently  larger  coeffi¬ 
cients  than  the  other  variables,  but  were  of  relatively  magnitude. 

As  an  estimate  of  the  efficiency  of  the  background  nnd  attitudinal 
variables  as  predictors  of  performance  ratings,  the  correlation  of  the 
optimally  weighted  set  of  variables  listed  in  Table  3  was  correlated 
against  Item  10  (on  overall  evaluation)  ratings  by  peers  in  group  P-2. 
The  R  was  .23*  Considering  that  this  is  a  back  validity  and  typical 
of  R 1 8  against  other  items  as  criteria,  the  practical  value  of  these 
background  and  attitudinal  variables  for  predicting  performance  ratings 
seems,  at  best,  limited. 


CONCLUSIONS 

From  the  analysis  of  ratings  in  this  study,  differences  between 
supervisor  and  peer  ratings  appear  to  be  negligible.  Background, 
attitudinal,  and  rater  observation  variables  were  found  to  have  limited 
predictive  effectiveness.  Other  than  a  general  factor  and  rater  factors 
three  dimensions  appeared  in  the  analysis:  Job  knowledge,  military 
behavior,  and  interpersonal  relations.  More  controlled  and  intensive 
study  is  needed  to  obtain  Information  on  a  variety  of  problems  in  the 
use  of  rating  data  os  assessment  of  Job  performance. 
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APPENDIX  A 


Rating  Items 


1.  BOW  DOCS  HE  (ST  ALONG  WHS  DDE  PBQPLE  HE  HAS  TO  WORK  WITH? 

A.  He  is  quite  popular  with  them  . 

B.  He  gets  along  reasonably  well  with  them. . . . 

C.  He  Is  quite  unpopular  with  them . 


2.  BOM  IMPORTANT  IS  HE  TO  HIS  UNIT? 

A.  It  would  be  hard  to  get  a  replacement  for  him  who  would  be  as 

good  as  he  Is..... . . . 

B.  He  could  be  easily  replaced . . . 

C.  The  unit  would  be  better  off  without  him . 


3.  BOW  WELL  DOES  HE  MASTER  NEW  DUTIES? 

A.  He  learns  things  fast . 

B.  He  keeps  up  with  the  group . 

C.  He  learns  things  slowly . 


4.  HOW  PRODUCTIVE  IS  HE? 

A.  He  does  more  than  his  share  of  the  work . - 

B.  He  does  no  more  and  no  less  than  is  required  of  him. 

C.  It  Is  difficult  to  get  any  work  out  of  him... . . 


5.  HOW  WELL  DOES  HE  KNOW  HIS  JOB? 

A.  He  knows  everything  about  his  Job  that  he  has  to  know . 

B.  He  Is  a  little  weak  on  some  of  the  things  he  ought  to  know. . . 

C.  He  has  a  lot  to  learn  to  reach  a  sutic factory  level.... . 


6.  BOW  MUCH  SUPERVISION  DOES  HE  NEED? 

A.  He  can  be  depended  on  to  do  all  his  work  without  supervision. 

B.  He  needs  some  supervision . . . 

C.  He  has  to  be  watched  all  the  time . . . . . 


7.  BOW  CAREFUL  IS  HE  IN  HIS  WORK? 

A.  He  doesn't  make  any  mistakes . 

B.  He  makes  some  mistakes*  but  nothing  serious 

C.  He  Is  very  careless  and  sloppy  In  his  work. 


8.  HOW  WELL  IS  HE  QUALIFIED  FOR  HIS  JOB? 

A.  Very  well . 

B.  Reasonably  veil . . . . . . . . 

C.  He  doesn't  have  the  necessary  skills  and  abilities  for  his  job. 


9.  IN  GENERAL,  HOW  IS  HE  GETTING  ALONG  ON  HIS  JOB? 

A.  Very  veil  for  a  man  of  his  experience... . 

B.  As  veil  as  can  be  expected  of  a  man  of  his  experience 

C.  He  is  not  working  up  to  expectations  for  a  man  of  his 

experience . . . . . . . 


10.  CONSIDERING  EVERYTHING,  BOW  WOULD  YOU  RATE  HIM  ON  HIS  USEFULNESS 
TO  THE  ARMY  ON  HIS  PRESENT  JOB? 

A.  HE  IS  THE  VERY  BEST  TYPE  OF  WORKER . 

He  is  above  average  in  every  way  as  a  worker  and  is  superior 
in  most  ways. 

B.  HE  IS  AN  OUTSTANDING  TYPE  OF  WORKER . 

He  is  very  good  in  most  ways  as  a  worker. 

C.  HE  IS  A  GOOD  TYPE  OF  WORKER . 

He  takes  his  work  seriously  and  does  the  best  he  can  at  all 
times. 

D.  HE  IS  AN  ACCEPTABLE  TYPE  OF  WCWCER . 

He  doesn't  always  do  the  best  he  can,  but  he  has  no  serious 
weaknesses  as  a  worker. 

E.  HE  IS  AN  UNSATISFACTORY  TYPE  OF  WORKER . 

He  seems  to  be  out  of  place  on  his  Job. 

F.  HE  IS  THE  VERY  WORST  TYPE  OF  WORKER . 

He  is  definitely  out  of  place  on  his  Job. 


11.  BOW  DOES  HE  COMPARE  WITH  OTHERS  IN  THE  UNIT  IN  GENERAL  MILITARY 
ACTIVITIES  SUCH  AS  MARCHING,  INSPECTIONS,  AND  BIVOUACS? 

A.  He  is  one  of  the  best  in  the  unit . . 

B.  He  gets  along  all  right . . . 

C.  He  is  not  very  good  in  this  respect . . 


12.  BOW  WOULD  HE  COMPARE  WITH  OTHERS  IN  THE  UNIT  IF  HE  HAD  TO  TAKE  THE 
LEAD  IN  AN  EMERGENCY? 

A.  Very  well . 

B.  About  the  same  as  most.. . . 

C.  Rot  very  well . . . . . . 


13.  BOW  WOULD  YOU  RATE  THIS  MAN  ON  HIS  MILITARY  APPEARANCE  AND  BEARING 
IN  COMPARISON  WITH  OTHERS  IN  THE  UNIT? 

A.  He  creates  a  good  Impression.... . . 

B.  He  creates  a  so-so  impression . . 

C.  He  creates  a  poor  ispresslon . . 
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Ik.  HOW  RELIABLE  AMD  TRUSTWORTHY  IS  BBT 

A.  You  can  depend  on  what  he  says  at  all  tines . 

B.  You  have  to  cheek  up  on  him  once  in  a  while...... 

C.  You  can't  depend  on  what  he  says  most  of  the  tine 


15.  WHAT  ERECT  DOES  BE  HAVE  OB  THE  UHXT? 

A.  Be  has  a  good  effect  on  the  unit . . 

B.  Be  doesn't  affect  the  unit  one  way  or  another 

C.  Be  has  a  bad  effect  on  the  unit . . . . . 


16.  CONSIDERING  EVERYTHING,  HOW  WOULD  YOU  RATE  THIS  HAN  GENERALLY  ON  HIS 
QUALITIES  AS  A  SOLDIER? 

A.  HE  IS  THE  VERY  BEST  TYPE  OF  XIDXER . 

Be  rates  high  on  all  the  qualities  of  a  soldier. 

B.  BE  IS  AN  OUTSTANDING  TYPE  OF  SOLDIER . 

Be  rates  high  on  most  of  the  qualities  of  a  soldier. 

C.  BE  IS  A  GOOD  TYPE  OF  SOLDIER . 

Be  takes  his  military  duties  seriously  and  tries  to  live 
lip  to  them  at  all  times. 

D.  HE  IS  AN  ACCEPTABLE  TYPE  OF  SOLDIER . 

He  doesn't  take  his  military  duties  seriously  at  all  times, 
hut  has  no  serious  weaknesses  as  a  soldier. 

E.  HE  IS  AN  UNSATISFACTORY  TYPE  OF  SOLDIER . 

Be  seems  to  be  out  of  place  in  the  Army. 

F.  HE  IS  THE  VERY  WORST  TYPE  OF  SOIDIER . 

He  doesn't  belong  in  the  Army  at  all. 


17.  CONSIDERING  EVERYTHING,  HOW  WOULD  YOU  RATE  THIS  MAN  ON  HIS  GENERAL 
VALUE  TO  THE  ARMY  IF  HE  DECIDED  TO  BE  AN  ARMY  CAREER  MAN? 

A.  HE  WOULD  BE  THE  VERY  BEST  TYPE  OF  CAREER  MAN . . 

He  would  be  an  asset  to  the  Army  in  every  respect. 

B.  HE  WOULD  BE  AN  OUTSTANDING  TYPE  OF  CAREER  MAN . 

He  would  be  an  asset  to  the  Army  in  most  respects. 

C.  HE  WOUID  BE  A  GOOD  TYPE  07  CAREER  MAN . 

He  would  have  something  to  offer  to  the  Army  and  would  be 
difficult  to  replace. 

D.  BE  WOULD  BE  A  SATISFACTORY  TYPE  OF  CAREER  MAN . 

Be  would  have  something  to  offer  to  the  Army  but  could  be 
easily  replaced. 

E.  BE  WOULD  BE  AN  UNSATISFACTORY  TYPE  OF  CAREER  MAN . 

He  would  have  very  little  to  offer  to  the  Army. 

F.  HE  WOUID  BE  THE  VERY  WORST  TYPE  OF  CAREER  MAN . 

He  would  do  more  harm  than  good  in  the  Amy. 
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SUMMRT  (ST  FACTOR  AIALTSXS 


Table  2 


81  54  34  05  "08  05  01  03  08  07  35  *09  56 

82  58  40  05  05  05  “03  00  02  0}  ^  JI  60 

S3  55  43  52  12  15  060456565855  55 

84  57  41  01  00  01  01  01  06  59  02  10  52 

85  45  30  U5  25  41  01  00  04  01  03  55 

86  46  33  "56  11  32  tl  04  06  54  03  Ol  45 

87  38  21  01  12  25  07  02  01  02  06  13  29 

88  55  51  51  16  38  51  07  53  07  01  22  63 

89  59  42  01  03  11  00  55  55  00  11  03  56 

810  65  47  "51  04  15  03  12  14  00  63  04  71 

311  49  26  00  56  06  04  25  35  04  09  56  53 
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816  66  48  01  06  04  07  24  38  10  03  01  89 

817  65  48  02  5  4  00  16  20  31  06  02  15  84 
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